Hive virtual store: moczarr-backed /gridlook/hive/ (phase 6d of issue #1) by espg · Pull Request #6 · espg/gridlook

espg · 2026-07-21T01:12:52Z

Refs #1 (phase 6d)

The reserved /gridlook/hive/ namespace becomes real: a moczarr-backed virtual store that serves an open_hive() product/AOI/window selection as ONE flat zarr v3 store the browser's unmodified zarrita reads. This is the cryocloud-MVP slice from the MVP map: post-englacial/zagg#314 stores are morton-only, so moczarr's fabricated NESTED cell_ids (fabricate_cell_ids="auto") is what makes them renderable at all — it rides every view served here.

What this adds

GET /gridlook/hive/open?store=<url>[&product=…][&aoi=<decimal,csv>][&window=…] — creates (or LRU-refreshes) a session-scoped view, returns {view, url, cells, cell_order, cached}; the url is the zarr source to paste into gridlook.
GET /gridlook/hive/<view-id>/<zarr-key> — serves the view's zarr objects (metadata documents + whole small chunks; no Range needed).
Auth posture inherited from phase 4: @web.authenticated everywhere, s3:// stores gated by the same GridlookProxy.allowed_buckets allowlist (403 otherwise), local paths only behind the new allow_local_hive_stores dev flag.
New traits: hive_max_views (LRU bound, default 8), hive_max_cells (per-view bound, default 500 000 — a pointed 413 tells you to narrow the AOI), allow_local_hive_stores.
gridlook-jupyter[hive] extra carrying moczarr @ git+https://github.com/espg/moczarr (until PyPI; tool.hatch.metadata.allow-direct-references gates the direct reference). The base install stays moczarr-free; hive.py imports it lazily and the open endpoint 500s with a pointed install hint when it's absent.

Materialize-on-open (the implementation lean, justified)

open_hive(...) → ds.to_zarr(zarr.storage.MemoryStore(), zarr_format=3, consolidated=False) per view, LRU-bounded. Chosen because: (a) views are AOI-scale and bounded by hive_max_cells, so the memory cost is capped and known (hive_max_views × hive_max_cells); (b) xarray writes the store, so the serve path contains zero zarr chunk/codec arithmetic — the handler serves opaque objects via the store's async get; (c) every request after open is a dict lookup. Chunks are written uncompressed (compressors: None): objects go whole over hub-local HTTP, and codec-free bytes are trivially predictable (the tests compare them raw). A streaming/virtual encoding — computing zarr objects on demand from the open dataset — is documented in hive.py as the future optimization if views outgrow memory; the URL contract wouldn't change.

View ids are deterministic (hash of the request tuple), so re-opening the same selection refreshes the existing view instead of duplicating it; serving objects LRU-bumps the view, so actively-rendered views survive eviction pressure.

Pre-6c compatibility shim (explicit, temporary)

The served root attrs do not advertise the stored morton convention block ({name: "morton", coordinate: "morton"}, mortie spec §5) — gridlook's gridTypeDetector.ts rejects any dggs.name != "healpix" today. Instead the view serves the healpix-compatible block the detector and Healpix.vue read now: name: "healpix", refinement_level (→ nside), coordinate: "cell_ids". That is honest at the data level — the fabricated NESTED ids are plain HEALPix NESTED indices at that refinement level, bit-for-bit what the existing sparse limited-area path consumes — so the MVP renders without phase 6c. The shim is marked in hive.py (_shim_dggs_attrs) and goes away when 6c lands the morton detector branch.

Tests (50 pass locally; hive suite skips without moczarr)

pytest-jupyter, mirroring the phase-4 suite, against moczarr's committed SERC fixture (zagg-written, morton-only) as a local-path store:

open → entry URL + cells; zarr.json carries the shimmed healpix attrs block
fabricated cell_ids chunk byte-equals serc_cell_ids_golden.npy (the last dual-written store's stored NESTED ids — fabricate-at-serve loses nothing)
AOI subsetting (one shard → 16 cells; ids match fabricate_cell_ids of the served morton words)
403 non-allowlisted bucket, 403 local-path when disabled, 404 unknown/evicted view, 404 missing object, 400 bad aoi/window, 413 over hive_max_cells, LRU eviction incl. refresh-bumps-recency
fixture discovery: moczarr repo checkout next to the installed package (or GRIDLOOK_MOCZARR_TESTDATA); plain [test] envs skip the module wholesale — existing CI is untouched and stays green

Gates run: pytest -q → 50 passed (env: uv venv && uv pip install -e '.[test]' -e /Users/espg/software/moczarr); plain [test]-only env → 35 passed, 1 module skipped; ruff check + ruff format --check clean.

MVP recipe (documented in jupyter/README.md)

pip install <wheel> "gridlook-jupyter[hive]" on the hub image/user env
c.GridlookProxy.allowed_buckets = ["my-zagg-outputs"] (or GRIDLOOK_ALLOWED_BUCKETS), restart the server
GET …/gridlook/hive/open?store=s3://my-zagg-outputs/store-root&aoi=4331422
open …/gridlook/, paste the returned view url as the dataset source — renders through the existing sparse-HEALPix path

Questions for review

CI: .github/workflows/jupyter.yml installs only [test], so the hive suite skips there (left untouched per the no-CI-edits convention). If you want it exercised in CI, the job needs a moczarr checkout + editable install step — say the word and it lands as a follow-up.
hive_max_cells default (500 000) is sized as ~40–50 MB/view at SERC-like variable counts; happy to retune.
The [hive] extra's git pin is unversioned (@ git+… HEAD); once moczarr tags/publishes, it becomes a normal moczarr>=X pin and allow-direct-references drops.

🤖 Generated with Claude Code

https://claude.ai/code/session_01N32Mx4rWhbTK8xXm7Mt2ax

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N32Mx4rWhbTK8xXm7Mt2ax

espg · 2026-07-21T01:24:44Z

+            "local-path hive stores are disabled — set "
+            "GridlookProxy.allow_local_hive_stores = True (development only)",
+        )
+    root = store_url.rstrip("/")


🤖 from Claude (review)

Unconstrained local-path oracle behind the dev flag. With allow_local_hive_stores=True, store= reaches any local path — probed /etc, /etc/passwd, ~/.jupyter all get past the gate and into open_hive(). Two problems on a multi-user hub:

The 400/404 bodies echo moczarr's messages verbatim ({e} at L#L245/L250-ish), which distinguish '/etc/passwd' is not a directory (file exists) vs no morton_hive.json at /etc (dir exists, no manifest) vs a missing path → a filesystem existence/type oracle for an authenticated user.

Any real hive store anywhere on the box (another user's $HOME, a shared scratch dir) is openable and servable — cross-user read, not just a dev convenience.

Default-off + documented-dev-only mitigates, but it's a full local-file reach the moment the flag flips. Suggest a rooted-directory trait (an allowlist of permitted roots + os.path.realpath containment check) so "dev" can't silently become whole-filesystem access. Probed with the test client; not blocking, but worth a rooted variant.

🤖 from Claude
Replaced the boolean with a rooted-directory trait local_hive_store_roots (empty = disabled): a local path is now accepted only when its os.path.realpath is contained in a configured root, and anything outside gets a single uniform 403 ("not within an allowed root") with no existence/type-distinguishing body, closing the filesystem oracle — d6dd5f2.

espg · 2026-07-21T01:24:45Z

+    level = dggs.get("refinement_level")
+    if level is None:
+        level = ds.attrs["morton_hive"]["cell_order"]
+    dggs.update({"name": "healpix", "refinement_level": int(level), "coordinate": "cell_ids"})


🤖 from Claude (review)

refinement_level is taken from cell_order with no float64-exact / point-kind guard. The frontend computes nside = 2**refinement_level (Healpix.vue L185) and does all cell math in JS doubles. But moczarr's fabricate_cell_ids clips POINT-kind words to order 24 (FLOAT64_EXACT_MAX_ORDER) and only warns for AREA cells above 24 — and that warning is emitted on the executor thread, so it's swallowed, never reaching the operator or the 200 response. Consequences the fixture (order 8) never exercises:

cell_order > 24 area store: served cell_ids are exact u64 the browser reads as lossy Numbers, and 12*nside*nside overflows 2^53 → silently wrong render, HTTP 200, no signal.

point-kind store where cell_order ≠ 24: shim advertises refinement_level = cell_order (e.g. 29) while the fabricated ids are clipped to order 24 → declared nside ≠ served ids' order → cells placed at the wrong scale.

Since this store exists only to feed a float64 browser consumer, recommend guarding cell_order > 24 at open (pointed 400/413) rather than serving an unrenderable/mismatched view. At minimum, document the ceiling here.

🤖 from Claude
The shim now derives refinement_level from the SERVED ids (point words clip to order 24, so a point store declares 24 and renders correctly instead of at a mismatched nside=2**29), and area stores whose ids exceed the float64-exact range (order > 24, or any id ≥ 12·4²⁴) are rejected at /open with a pointed 422 rather than served — 0ad2923.

espg · 2026-07-21T01:24:46Z

+        store_url: str, product: str | None, window: str | None, aoi: tuple[str, ...] | None
+    ) -> str:
+        payload = json.dumps(
+            [store_url, product, window, list(aoi) if aoi else None], separators=(",", ":")


🤖 from Claude (review)

View id isn't normalized → trivial LRU blowup / redundant materialization. Probed against the SERC fixture:

aoi=4331421,4331422 vs aoi=4331422,4331421 → different view ids, identical 32 cells

aoi=4331421,4331421,4331422 (dup) → a third id, same 32 cells

store=…/serc_hive vs …/serc_hive/ (trailing slash) → different ids, same store

Every distinct id is a full re-open (S3 GETs + concat) and consumes an LRU slot, so a client varying token order / duplicates / trailing slash multiplies work and evicts legitimately-open views — an easy way to thrash the cache out from under other users. build_view already rstrips the root for the data path, but the hash keys on the raw store_url + raw aoi tuple. Canonicalize before hashing: rstrip store_url, and sorted(set(int(t) for t in aoi)) so order/dupes collapse to one id.

🤖 from Claude
view_id now canonicalizes before hashing — rstrip the store, normalize empty params to absent, and sorted(set(int(t) ...)) the aoi — so reordered/duplicated tokens and a trailing slash all collapse to one id (unit-tested) — 89d9fa7.

espg · 2026-07-21T01:24:47Z

+                max_cells=proxy.hive_max_cells,
+            )
+            try:
+                view = await IOLoop.current().run_in_executor(None, build)


🤖 from Claude (review)

The advertised memory bound covers resident views only; concurrent in-flight builds are uncapped. Real footprint on the fixture is 52 B/cell → 26 MB per 500k-cell view, ~208 MB across 8 (so the config's ~100 B/cell is a safe 2× over-estimate — good). But build_view is dispatched to the default ThreadPoolExecutor (run_in_executor(None, …), up to ~min(32, cpu+4) threads) with no concurrency limit, and each in-flight build holds the xarray ds plus the MemoryStore copy transiently. N concurrent large-view opens ⇒ N×(ds+store) on top of the 208 MB resident cache — so "hive_max_views × hive_max_cells bounds the cache footprint" (config help + PR body) understates peak on a 2–4 GB pod under a burst of opens. Consider a small bounded executor or a semaphore around opens, or at least document that the bound is resident-only.

🤖 from Claude
Added a hive_max_concurrent_builds trait (default 2) and a per-loop module asyncio.Semaphore around the executor offload, so a burst of opens queues rather than holding N×(ds+store) transiently on top of the resident cache; the config help now says the cell bound is resident-only, and a test opens 4 at once and all succeed — 27862be.

espg · 2026-07-21T01:24:47Z

+    dggs.update({"name": "healpix", "refinement_level": int(level), "coordinate": "cell_ids"})
+    dggs.setdefault("spatial_dimension", "cells")
+    ds.attrs["dggs"] = dggs
+    ds.attrs.setdefault("zarr_conventions", [_DGGS_CONVENTION_ENTRY])


🤖 from Claude (review)

Shim rewrites the dggs block to healpix but leaves the stored morton-dggs entry in zarr_conventions. Probed the served root zarr.json: zarr_conventions carries both the generic dggs entry and the stored morton-dggs ("Packed-u64 morton (HEALPix) DGGS convention") entry, while the dggs block now says name: healpix, coordinate: cell_ids. Today's detector only checks that the zarr_conventions key exists then reads the dggs block (gridTypeDetector.ts determineGridTypeFromZarrConvention), so this is harmless now — but the served store is internally contradictory: it claims conformance to the morton-dggs convention while serving a healpix block against a cell_ids coordinate. A stricter/future validator that keys on the convention entries would trip. Since the shim already owns the block rewrite, consider dropping (or replacing) the morton-dggs entry here too so the envelope and the block agree.

🤖 from Claude
The shim now drops the stored morton-dggs entry from zarr_conventions (keeping the generic dggs registry entry) so the served envelope and the healpix-flavored block agree, and preserves the store’s own morton block verbatim under a private _gridlook_source_dggs attr for provenance — 1077815.

espg · 2026-07-21T01:25:13Z

🤖 from Claude (review)

Fresh-context adversarial review of phase 6d (the moczarr-backed /gridlook/hive/ virtual store). Set up the worktree env per the PR body (uv pip install -e '.[test]' -e /path/to/moczarr) — 50 pass; all probes below were run against the SERC fixture via the test client.

Inline findings (severity-ordered)

allow_local_hive_stores is an unconstrained local-path oracle (medium) — reaches /etc, /etc/passwd, ~/.jupyter; error text distinguishes file/dir/missing; any user's on-box hive store is readable. Default-off/dev-only mitigates. Suggest a rooted-directory trait.
Shim refinement_level = cell_order with no >24 / point-kind guard (medium) — nside = 2**refinement_level in JS doubles; fabricate_cell_ids clips point words to order 24 and only warns (swallowed on the executor) for area cells >24. cell_order>24 or point-kind ⇒ silently wrong/mismatched render at HTTP 200. Fixture is order 8, untested.
View id not normalized (low-med) — probed: reordered aoi, duplicate aoi, and trailing-slash store each mint a distinct view id for identical data → redundant materialization + LRU thrash that evicts other users' views.
Memory bound is resident-only; concurrent builds uncapped (low-med) — real 52 B/cell → 26 MB/view, 208 MB @8 (config's ~100 B/cell is a safe 2× over-estimate). But run_in_executor(None, …) has no concurrency cap and each build holds ds+store transiently, so peak can exceed the advertised bound under a burst.
Shim leaves stored morton-dggs in zarr_conventions while the dggs block is rewritten to healpix (low) — served envelope is internally contradictory; harmless to today's detector (verified it only checks the key then reads the block), trips a stricter validator.

Minor (not posted inline)

_parse_aoi doesn't cap token count — 5000-token aoi= accepted (200); output is gated by hive_max_cells so impact is a mild input-side DoS (regex + moc_and over the list). A token-count cap would close it.
The 413 body suggests narrowing aoi= but not window=.

Verified sound

Auth: @web.authenticated on both handler families; no-token requests serve no data (probed — request never completes at the gate, never 200).
S3 allowlist enforced on store= (probed s3://sneaky-bucket → 403); non-s3:// scheme → 403; casing can't bypass (only lowercase s3:// hits the allowlist branch).
No path traversal: product regex forbids / and a leading dot, so it can't escape the root; the <key> capture is a dict lookup in the MemoryStore, so ../… just 404s (no filesystem touch).
Event-loop nesting is fine: build runs on an executor thread (no running loop → moczarr/zarr use their own background loop via sync()); serve awaits MemoryStore.get on the tornado loop (no loop affinity). Probed 4 simultaneous opens → distinct views, correct cells, no deadlock/cross-talk.
Size gate fires on ds.sizes before any data load → 413 before materialize.
LRU has no data race: get/put/serve all run on the single tornado loop; only build_view is offloaded and cache.put happens back on-loop.
Uncompressed byte-equality of served chunks holds (the raw-compare tests are honest).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N32Mx4rWhbTK8xXm7Mt2ax

…(issue #2) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N32Mx4rWhbTK8xXm7Mt2ax

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N32Mx4rWhbTK8xXm7Mt2ax

espg · 2026-07-21T01:40:19Z

🤖 from Claude

Acknowledging the two minor (non-inline) notes from the review summary:

_parse_aoi token-count cap — not added as a hard cap, but the view-id canonicalization from finding (3) now sorted(set(...))-dedupes aoi tokens, so repeated tokens collapse before hashing/materialization (a duplicate-driven blowup no longer mints distinct views). A true count cap on distinct tokens is still open; leaving it for @espg since output is already gated by hive_max_cells (mild input-side cost only).
413 body naming only aoi= — addressed while in finding (2): the oversize message now reads "narrow the aoi= or window= selection" (27862be region / the 413 branch), so both narrowing levers are named.

espg · 2026-07-21T06:06:42Z

If you want it exercised in CI, the job needs a moczarr checkout + editable install step — say the word and it lands as a follow-up.

this would be nice, and is easier now that moczarr's on pypi...

espg and others added 3 commits July 20, 2026 18:11

feat: moczarr-backed hive virtual store under /gridlook/hive/ (phase 6d)

dbd78e0

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N32Mx4rWhbTK8xXm7Mt2ax

test: hive virtual-store suite against the moczarr SERC fixture

ffbaa90

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N32Mx4rWhbTK8xXm7Mt2ax

docs: hive endpoints and cryocloud MVP recipe in jupyter README

4d5a679

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N32Mx4rWhbTK8xXm7Mt2ax

espg commented Jul 21, 2026

View reviewed changes

espg and others added 5 commits July 20, 2026 18:33

fold review: rooted local-hive-store roots, not a boolean (issue #1)

d6dd5f2

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N32Mx4rWhbTK8xXm7Mt2ax

fold review: derive refinement_level from served ids, guard order>24 …

0ad2923

…(issue #2) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N32Mx4rWhbTK8xXm7Mt2ax

fold review: canonicalize the hive view id before hashing (issue #3)

89d9fa7

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N32Mx4rWhbTK8xXm7Mt2ax

fold review: bound concurrent hive builds with a semaphore (issue #4)

27862be

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N32Mx4rWhbTK8xXm7Mt2ax

fold review: serve one coherent dggs convention block (issue #5)

1077815

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N32Mx4rWhbTK8xXm7Mt2ax

espg marked this pull request as ready for review July 21, 2026 01:41

espg merged commit 1a7c9bf into main Jul 21, 2026
3 checks passed

espg deleted the claude/1-phase6d-hive-store branch July 21, 2026 06:06

espg mentioned this pull request Jul 21, 2026

Plan: shard-vector + output-grid globe viewer, packaged for JupyterHub (cryocloud) #1

Open

6 tasks

Conversation

espg commented Jul 21, 2026

What this adds

Materialize-on-open (the implementation lean, justified)

Pre-6c compatibility shim (explicit, temporary)

Tests (50 pass locally; hive suite skips without moczarr)

MVP recipe (documented in jupyter/README.md)

Questions for review

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

espg Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

espg commented Jul 21, 2026

Inline findings (severity-ordered)

Minor (not posted inline)

Verified sound

Uh oh!

espg commented Jul 21, 2026

Uh oh!

espg commented Jul 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

espg Jul 21, 2026 •

edited

Loading